A storage system is a computer that provides storage service relating to the organization of information on writeable persistent storage devices, such as memories, tapes or disks. The storage system is commonly deployed within a storage area network (SAN) or a network attached storage (NAS) environment. When used within a NAS environment, the storage system may be embodied as a file server including an operating system that implements a file system to logically organize the information as a hierarchical structure of directories and files on, e.g. the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information, such as the actual data for the file. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
The file server, or filer, may be further configured to operate according to a client/server model of information delivery to thereby allow many client systems (clients) to access shared resources, such as files, stored on the filer. Sharing of files is a hallmark of a NAS system, which is enabled because of semantic level of access to files and file systems. Storage of information on a NAS system is typically deployed over a computer network comprising a geographically distributed collection of interconnected communication links, such as Ethernet, that allow clients to remotely access the information (files) on the file server. The clients typically communicate with the filer by exchanging discrete frames or packets of data according to pre-defined protocols, such as the Transmission Control Protocol/Internet Protocol (TCP/IP).
In the client/server model, the client may comprise an application executing on a computer that “connects” to the filer over a computer network, such as a point-to-point link, shared local area network, wide area network or virtual private network implemented over a public network, such as the Internet. NAS systems generally utilize file-based access protocols; therefore, each client may request the services of the filer by issuing file system protocol messages (in the form of packets) to the file system over the network. By supporting a plurality of file system protocols, such as the conventional Common Internet File System (CIFS), the Network File System (NFS) and the Direct Access File System (DAFS) protocols, the utility of the filer may be enhanced for networking clients.
A SAN is a high-speed network that enables establishment of direct connections between a storage system and its storage devices. The SAN may thus be viewed as an extension to a storage bus and, as such, an operating system of the storage system enables access to stored information using block-based access protocols over the “extended bus”. In this context, the extended bus is typically embodied as Fibre Channel (FC) or Ethernet media adapted to operate with block access protocols, such as Small Computer Systems Interface (SCSI) protocol encapsulation over FC (FCP) or TCP/IP/Ethernet (iSCSI). A SAN arrangement or deployment allows decoupling of storage from the storage system, such as an application server, and some level of storage sharing at the application server level. There are, however, environments wherein a SAN is dedicated to a single server. When used within a SAN environment, the storage system may be embodied as a storage appliance that manages data access to a set of disks using one or more block-based protocols, such as FCP.
One example of a SAN arrangement, including a multi-protocol storage appliance suitable for use in the SAN, is described in U.S. patent application Ser. No. 10/215,917, entitled MULTI-PROTOCOL STORAGE APPLIANCE THAT PROVIDES INTEGRATED SUPPORT FOR FILE AND BLOCK ACCESS PROTOCOLS by Brian Pawlowski et al., now published as U.S. Patent Publication No. 2004/0030668 A1 on Feb. 12, 2004.
It is advantageous for the services and data provided by a storage system, such as a storage system, to be available for access to the greatest degree possible. Accordingly, some storage systems provide a plurality of storage systems organized as a storage system cluster, with a property that when a first storage system fails, the second storage system is available to take over and provide the services and the data otherwise provided by the first storage system. When the first storage system fails, the second storage system in the cluster (the “partner”) assumes the tasks of processing and handling any data access requests normally processed by the first storage system. One such example of a storage system cluster configuration is described in U.S. patent application Ser. No. 10/421,297, entitled SYSTEM AND METHOD FOR TRANSPORT-LEVEL FAILOVER OF FCP DEVICES IN A CLUSTER, by Arthur F. Lent, et al., now issued as U.S. Pat. No. 7,260,737 on Aug. 21, 2007. In such a storage system cluster, an administrator may desire to take one of the storage systems offline for a variety of reasons including, for example, to upgrade hardware, etc. In such situations, it may be advantageous to perform a “voluntary” user-initiated takeover operation, as opposed to a failover operation. After the takeover operation is complete, the storage appliance's data is serviced by its partner until a giveback operation is performed. As such, the terms “failover” and “takeover” may be used interchangeably.
During the takeover operation, the surviving storage appliance sets SCSI reservations on the disks normally serviced by the repaired storage appliance. These SCSI reservations prevent any other devices from accessing the disks. The use of SCSI reservations in a cluster failover operation is further described in U.S. patent application Ser. No. 10/086,657, entitled APPLIANCE AND METHOD FOR CLUSTERED FAILOVER WITHOUT NETWORK SUPPORT, by John A. Scott, now issued as U.S. Pat. No. 7,039,828 on May 2, 2006. Additionally, the surviving storage appliances sets an appropriate state in an on-disk mailbox signifying that it has taken over the repaired storage appliance. The on-disk mailbox is a known location on disks that is accessible by the storage appliances in a cluster for transmitting messages and status information during cluster operation. On-disk mailboxes are further described in U.S. patent application Ser. No. 10/378,400, entitled APPLIANCE AND METHOD FOR COORDINATING CLUSTER STATE INFORMATION, by Richard O. Larson, et al., now issued as U.S. Pat. No. 7,231,489 on Jun. 12, 2007.
During normal cluster operation, when a storage appliance fails, the surviving storage appliance performs a failover operation. After the failover, the surviving partner processes data access requests that were originally directed to both the surviving storage appliance and the failed storage appliance. An appliance administrator or user then repairs the failed (now repaired) storage appliance and begins the initialization, or boot, process for the repaired storage appliance. The repaired storage appliance detects a set of disk reservations on its disks and halts its initialization process. The administrator then executes a giveback command on the surviving storage appliance, which causes the disk reservations to be cleared. Once the giveback command is executed on the surviving storage appliance, the storage appliance stops processing data access requests directed to the repaired storage appliance. The repaired storage appliance then restarts its initialization procedure and begins processing data access requests.
A noted disadvantage of the prior art is that during the time from the initiation of the giveback command until the time when the repaired storage appliance has completed its boot process and initialized its disk subsystem, clients of the repaired storage appliance experience a loss of connectivity to data. This loss of connectivity to data may be in the tens of seconds. During this time, clients of the repaired storage appliance are not able to access their data serviced by the repaired storage appliance, nor may they complete write operations to store additional data.
Another noted disadvantage of the prior art is that if the repaired storage appliance is not functional when the giveback command is executed, a failover procedure may be initiated immediately after the surviving storage appliance had already ceased serving data access requests directed to the failed or repaired storage appliance. This can result in a further loss of data connectivity for clients of the storage appliance. Additionally, the repaired storage system may fail to reinitialize and the surviving storage system may not initiate another takeover application. In such an event, clients of the repaired (now failed) storage system are without data connectivity until the failed storage system is manually repaired and reinitialized.